Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GODRIVER-3156 Detect and discard closed idle connections. #1815

Merged
merged 5 commits into from
Oct 1, 2024

Conversation

matthewdale
Copy link
Collaborator

GODRIVER-3156

Summary

  • When checking if a connection is closed (either during check out or during the background maintenance routine), try to read from the connection to check if it's still alive.
    • Only check for the liveness of the connection if it's been idle for more than 10 seconds to prevent adding latency to most check-out operations.
  • Change connection.idleDeadline to connection.idleStart, allowing us to calculate different idle deadlines for different purposes.

Background & Motivation

Currently the Go driver never attempts to detect connections that were closed by the other side when checking out connections. As a result, it's possible to return a connection that's been closed by the other side where any read or write operations will immediately fail. Instead, we should check if the connection has been closed when checking out a connection.

@mongodb-drivers-pr-bot mongodb-drivers-pr-bot bot added the priority-3-low Low Priority PR for Review label Sep 16, 2024
x/mongo/driver/topology/connection.go Outdated Show resolved Hide resolved
// If the connection has been idle for less than 10 seconds, skip the liveness
// check.
idleStart, ok := c.idleStart.Load().(time.Time)
if !ok || idleStart.Add(10*time.Second).After(time.Now()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the significance of 10 seconds?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My reference was the PyMongo driver, which checks connection liveness when it's idle for >1 second. I wanted to mitigate the risk that checking liveness too often would cause performance problems (since a liveness check takes at least 1ms), so I increased that threshold to 10s.

x/mongo/driver/topology/pool_test.go Outdated Show resolved Hide resolved
@blink1073 blink1073 added priority-1-high High Priority PR for Review and removed priority-3-low Low Priority PR for Review labels Sep 27, 2024
@blink1073
Copy link
Member

This looks relevant: x/mongo/driver/topology/pool_test.go:1230:4: undefined: noerr

Copy link
Contributor

API Change Report

No changes found!

prestonvasquez
prestonvasquez previously approved these changes Sep 30, 2024
Copy link
Collaborator

@prestonvasquez prestonvasquez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I recommend adding a unit test for isAlive:

func TestConnection_IsAlive(t *testing.T) {
	t.Run("idle time lt 10 seconds", func(t *testing.T) {
		mockConn := &drivertest.ChannelNetConn{}
		conn := &connection{
			nc: mockConn,
		}

		conn.idleStart.Store(time.Now())
		assert.True(t, conn.isAlive())
	})

	t.Run("idle time gt 10 seconds", func(t *testing.T) {
		mockConn := &drivertest.ChannelNetConn{
			ReadResp: make(chan []byte, 1),
		}

		mockConn.ReadResp <- []byte{5, 0, 0, 0, 0}

		conn := &connection{
			nc: mockConn,
		}

		conn.idleStart.Store(time.Now().Add(-11 * time.Second))
		assert.False(t, conn.isAlive())
	})

	t.Run("read error", func(t *testing.T) {
		mockConn := &drivertest.ChannelNetConn{
			ReadErr: make(chan error, 1),
		}

		mockConn.ReadErr <- errors.New("")

		conn := &connection{
			nc: mockConn,
		}

		conn.idleStart.Store(time.Now().Add(-11 * time.Second))
		assert.False(t, conn.isAlive())
	})

	// TODO:
	// case: deadline exceeded
}

@blink1073
Copy link
Member

drivers-pr-bot please backport to release/1.17

@matthewdale matthewdale merged commit 6952473 into mongodb:v1 Oct 1, 2024
30 of 33 checks passed
mongodb-drivers-pr-bot bot pushed a commit that referenced this pull request Oct 1, 2024
Co-authored-by: Steven Silvester <[email protected]>
(cherry picked from commit 6952473)
blink1073 pushed a commit that referenced this pull request Oct 1, 2024
matthewdale added a commit to matthewdale/mongo-go-driver that referenced this pull request Oct 1, 2024
matthewdale added a commit that referenced this pull request Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority-1-high High Priority PR for Review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants